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(57) ABSTRACT 

In a data communication system having network traffic with 
flows, a network traffic director system using router mod- 
ules. The router modules direct traffic based on optimizing 
a merit function or penalty function to reduce costs of 
congestion for stochastically changing demands and flows in 
the data communication system. The router modules 
exchange values with neighboring router modules. Based on 
the exchanged values and values local to a router module, 
flow conditions are checked and if necessary the local values 
are adjusted until the flow conditions are satisfied or a time 
period expires. Adjustments are associated with optimizing 
a merit function or penalty function. Based on the adjusted 
values, the router module adjusts parameters to be used to 
direct packets of the network traffic flows to other router 
modules or other destinations within the data communica- 
tion system. An aggregation scheme is used for reducing the 
number of values stored in a single router module. 

29 Claims, 7 Drawing Sheets 
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METHOD AND APPARATUS FOR NETWORK 
CONTROL 

TECHNICAL FIELD 

The present invention is related generally to data com- ^ 
munication systems and, in particular, to network control of 
data communication systems. 

BACKGROUND OF THE INVENTION 

Data communication systems generally consist of mill- 
tiple networks that are linked together with bridges, 
switches, routers, or other devices. These linking devices 
direct data packets along various paths from an origination 
to a destination. Since a data communication system tends to 
be large with many networks and interconnecting links, 
typically a data packet can take many paths from its origi- 
nation to its destination. A goal of a network traffic director 
system is to direct the journey of each data packet from its 
origination to its destination so that the capacity of the 
network is efficiently utilized. '^^ 

Traditional network traffic director systems utilize a plu- 
rality of network routers based on Internet Protocol (IP). The 
IP routing approaches are based on a Shortest Path (SP) 
method in which packets are sent to their destinations along ^ 
the shortest paths in the network. There are various methods 
to determine shortest paths among the IP routing 
approaches. However, these approaches all share a common 
assumption that if each packet travels along the shortest 
path, then the overall workload required for the routing will 
remain as small as possible. 

The assumption of traditional routing systems would hold 
true if the overall performance of a routing system could be 
adequately measured solely on the basis of the total number 
of decisions that routers need to perform. However, other 35 
factors are involved in the overall performance of a routing 
system. One of these factors is the amount of congestion in 
a data communication system. Some theoretical proposals 
exist which attempt to address the congestion of network 
traffic director systems. These approaches, however, are 
impractical in providing adequate solutions to the conges- 
tion problem. The end result of some of these approaches 
produces, on average, flows that are httle different than the 
shortest path methods of IP routing taking congestion into 
account. Another theoretical approach to congestion is 45 
intended for environments having predefined aspects, such 
as loading and network topology. However, a data commu- 
nication system is seldom sufficiently predictable for these 
predefined approaches. 

Other theoretical approaches attempt to address an aspect 50 
of the stochastic (random) nature of flows in a data com- 
munication system by allowing for randomness in the arrival 
times of individual data packets into a linking device. These 
approaches, however, are still formulated in a deterministic 
manner by requiring that demand or load rates and flow rates 55 
in other parts of the communication system be known in 
advance of their actual occurrence. 

These prior art deterministic approaches are not ideally 
suited for data communication system environments in 
which demands or loads and flows in other portions of the 60 
data communication system have stochastic distributions 
due to changing demands or loads of individual users or due 
to any multiplexing involved. This means that not only are 
the arrival times of individual packets random, but also the 
rates of the demands or loads and flows in other portions of 65 
the data communication system are random and not known 
before their actual occurrence. 
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An effective optimization method must not only minimize 
congestion in the data communication system but it must be 
robust enough to adapt to sudden changes in conditions of 
the data communication system such as packet arrival, 
loading on the system, and system topology without pre- 
defined scenarios of when or how these changes will occur. 

In data communication systems, loading changes quickly 
and dramatically due to such influences as changing 
demands or loads of individual users and multiplexing 
methods such as time division multiplexing (TDM) where 
time slots of system use are divided among individual users. 
With TDM, a data communication system constantly 
changes the particular users on the system at any given 
moment, thus loading changes rapidly. Network topology 
can also have rapid stochastic changes due to situations such 
as equipment failures or weather conditions. 

Also, whereas the prior art systems and methods focus on 
a single indicator, Such as shortest path or average delay, as 
a basis to optimize data flow, there are many additional 
factors. These include average queue length at various data 
communication system linking devices, variance or standard 
deviation in individual delays of data packets traveling from 
an origination to a destination, and average and variance of 
the individual utihzed capacity of links and hnking devices 
in a data communication system. Thus, a system and method 
is greatly desired which optimizes packet flow in a data 
communication system based on oumerous factors while 
being robust enough to effectively adapt to unexpected 
events, 

SUMMARY OF THE INVENTION 

The present invention is directed to a network traffic 
director system in a data communication system having 
network traffic with randomly distributed demands and 
flows. In one aspect of the present invention the network 
traffic director system includes router modules configured to 
direct data packets in the data communication system, each 
router module being a home router module including a 
neighborhood supervisor. The neighborhood supervisor is 
configured to send home potentials of the home router 
module to neighborhood supervisors of neighboring router 
modules and to receive neighbor potentials of the neighbor- 
ing router modules from neighborhood supervisors of neigh- 
boring router modules. The home router module further 
includes a dynamic load balancer configured to determine 
flows based on the home and neighbor potentials. The 
dynamic load balancer adjusts the home potentials if first 
conditions including flow conditions are not met. The 
dynamic load balancer also updates routing tables if second 
conditions based on the adjusted home potentials are met. 
The home router module also includes a dynamic data flow 
splitter configured to receive data packets from networks 
and router modules. The dynamic data flow splitter selects a 
portion of the data communication system for each data 
packet received based on the updated routing tables. Each 
received data packet is transmitted to the portion of the data 
communication system selected by the dynamic data flow 
splitter for the received data packet. 

Another aspect of the invention includes the home poten- 
tials of each home router module being associated with a 
pair of nodes of the data communication system including an 
origination node and a destination node. The home poten- 
tials of each router module are also associated with a quality 
of service level. The neighborhood supervisor is configured 
to send the home potentials to neighborhood supervisors of 
neighboring router modules when the flow conditions are 
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not met. The flow conditions include conservation of flows. are sent between the networks t20 via the router modules 

The dynamic load balancer is configured to determine flows 130. The network traffic director system 110 as embodied in 

by using a response function associated with optimizing at the router module 130 allows more efficient utilization of 

least one of a penalty function or a merit function for data communication systems over prior art linking devices, 

stochastic demands or flows of the data communication 5 The router module 130 is configured to adapt to sudden 

system. The dynamic data flow splitter is configured to changes in the data communication network, such as ran- 

selecl the portion of the data communication system for each ^omly and rapidly changing service demand or load rates or 

packet received further based on either a Markov protocol, Apology, while optimizing utflization of the data comrmi- 

a Routing Wheel protocol, previously selected portions of «y^^^^. ^"^^"^ numerous factors including traffic 

/'^ jj r -jj* „ delays, queue lengths, vanance in mdjvidual delays, average 

the data communication when an address for a received data lo . • c .-\- ^ r i- i j i- i • 

. , . , , , . . • .V J and variance of utiUzed capacity of links and linking 

packet IS a same type as an addre^ stored in the dynamic devices, nnmber of lost packet^, latencies, and throughput of 

data flow sputter, or on a quality of service level associated linking devices 

with a data packet. jj^^^ ^^^^ methods, for the depicted embodiment, 

In a further aspect of the invention the neighborhood the optimization methods \ised account for the numerous 

potentials are stored according to an aggregation scheme. 15 factors influencing utilization of a data communication 

The merit or penalty functions are at least locally approxi- system by formulating a penalty function (which accounts 

mated by a quadratic function of the flows. The ideal data for negative effects, such as lost packets, increased delays, 

flows are further based on resistance of arcs associated with high latency, diminished throughput, among other things) or 

the router module wherein each resistance of an arc is its opposite merit function (which accounts for beneficial 

configured to be a function of either arc capacity or a ^ effects). The merit and penalty functions are formulated for 

function of flows of flie arc. The neighbor router modules are optimization and determination of equilibrium points given 

configured to be associated with the home director module a rapidly changing, stochastic environment of random 

through at least one of a political, organizational, demands upon a data communication system, random flows 

topological, topical, or geographical relation. In a further l^ro^gh a data communication system and random changes 

aspect each router module is configured to adjust home 25 m topology of a data commumcaUon system 

potentials to an equilibrium point for a difference function This stochastic approach is far different from prior art 

associated with flows of arcs associated with the router mefliods, which rely on predictable, non-stochastic en 

module. The difference fiinction is associated with differ- ^^J^' f'^^ ^^u'^'u^'T fo^^^^^^^^^.^^^^ys^ems 

. ^ . * 1 J • Lu * 1 TV and methods used by the depicted embodiment of the 

encesbetweenhomepotentialsandneighborpotenials.ihe ^^^^^^^ ^^^^ ^^^^^^ 

ment function or penalty fiinc^ionmvolvescoste of a)nges- 30 ^^^^^^^^ implemented in a data communication system 

tion including at least one of traffic delays, high latency, ^^^^^^ j^^^.^g extraordinary processing requirements, 

diminished throughput, lost packets or unresponsiveness to resource needs or other demands upon the overall data 

sudden changes in topology or loading. communication system. These and other advantages will 

_ ^ ^ „,„.r^r, become apparent in the following detailed description. In the 

BRIEF DESCRIPTION OF THE DRAWINGS 35 following description, numerous specific details are pro- 

HG. 1 is a topology diagram representative of a network ^ided such as methods and systems used to collect infor- 

traffic director system of the present invention. f^^^^ ^he router modules 130 to determine whether 

^ , t.ij- L . L. conditions are satisfied, and to adjust values related to the 

HGS. 2A and 2B are block diagrams showing the mter- ^^^^^^ ^^^^^^^ accordingly. One skiUed in the relevant 

connection detail of router modules of the network traffic 40 however, wiU recognize that the invention can be 

director system of FIG. 1. practiced without one or more of the specific details, or with 

FIG. 3 is a block diagram showing internal detail of a other devices and methods. In other instances, well known 

router module of FIG. 2. structures or operations are not shown or described in detail 

FIG. 4 is a topology diagram showing internal detail of a to avoid obscuring the description of the embodimenU. 

router module of FIG. 2. order to connect the networks 120 of the data commu- 

- . r t. nication system together, the router modules 130 are con- 

FIG. 5 IS a relational diagram of an aggregation scheme .j* .u • sun u 

... , 1 . ? . . fi-Ti- 1 nected together via communication umts 210 as shown m 

used in the network traffic director system 01 FIG. 1. i-Fr^o j'^rt ti. • ** ha r 1 a 

^ FIGS. 2 A and 2B. The communication units 210 are unked 

FIG. 6 is a topology diagram showing internal detail of an together by data communication pathways 212. The com- 

arc of a router module of FIG. 2. munication units 210 in one embodiment are Ethernet net- 

FIG, 7 is a flowchart of a method used by the router work cards and the data communication pathways 212 are 

modules of FIG. 2. Ethernet cables. Ethernet is a term well known in the art 

referring to collision sense multiple access/coUision detec- 

DETAILED DESCRIPTION OF THE techniques as described under the IEEE 802 series 

INVENTION specifications. In other embodiments of the present 

Large communication data systems are comprised of a invention, the communication units 210 and pathways 212 
collection of networks linked together by linking devices, are other types of point-to-point data communication Unks. 
such as routers, bridges, and switches known herein as Someof the communication units 210 shown in FIGS. 2A 
router modules or director modules. The depicted embodi- and 2B are not shown to be connected via any data com- 
ment in FIG. 1 has a network traffic director system 110 in 60 munication pathway 212. These communication units 210 
a data communication system which links a coUection of can be connected to other router modules 130 which are not 
networks 120 via router modules 130 and associated links shown in FIGS. 2Aand 2B. The number of communication 
140 and links 150. A link 140 is a data communication path units 210 shown for each individual router module 130 is not 
for data packets to travel between a network 120 and a router intended to limit the depicted embodiment since some router 
module 130. A link 150 is a data communication path for 65 modules have more communication units 210 than shown in 
data packets to travel between two router modules 130. FIGS. 2A and 2B ff necessary to link additional router 
Network traffic is made up of individual data packets which modules. 
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Each router module 13(ia-d, as showa in FIGS. 2A and router module 130 to diflferent router modules 130 through 

2B, is connected to a data network 120a-<i. However, the one of the communication units 210 or to a directly linked 

number of data networks 120 connected to any one router network 120. Four different methods of determination, also 

module 130 varies, and some router modules 130 are only called splitting protocols, arc typically chosen, 

used to interconnect other router modules and arc not 5 Markovian splitting is the simplest of these routing pro- 

directly connected to any network 120. In other mstances, , 1 « • 1 . • r. t »u i^* 1 • .1 

, ^ , . Lj- ^1 1 V* tocols to unplement m software. In the Markovian protocol, 

router modules 130 can be directly connected to a plurality f ui -^-lo * r *l j ■ j ^ • * 

of networks 120. Each router module 130 has a neighbor- ""'-"f '^"^ ^18 transforms Uie desued flows mto 

hood supervisor unit 214 that includes a values exchanger, equivalent routing fracUons^ Only the flows out of the router 

a dynamic load balancer unit 216, a routing table unit 218 f °dule 130 are relevant For each destmation if is the 

with a pluraHty of routing tables, and a dynamic data flow '° '^^''^ ° *f ' ^ the neighbors j of the 

Splitter unit 220 router module 130, then the fraction of the flow to this 

m LLv. • '.^^ATi. . neighbor is p,^f 72, -f. In turn, the dynamic data flow splitter 

The neighborhood supervisor unit 214 of a home router ■ \u^c u l-v* e i,/ i 

, , , , J i . umt 220 views the fraction as a probability of a Markov 

module 130 exchanges information with router modules that ^. , ^ i * *i. j ■ j * ii 

. , , ^ ^. ^ ^ J 1 ■ i_L - process. When routmg a data packet, the dynamic data flow 

are neighbors to the home router module. These neighbormg i<- ^-^^ 'y%g\ t ^ *u u u i%- — ^* * 

, , , ^-rt J- . J . . splitter unit 220 looks up the probabiUties correspondmg to 

router modules 130 are directly connected to the home i * j n o v** 4U 

, - „ . ^ / r^^„ J *u the packet destination, Pi, p2, Pjt. The flow splitter then 

router module. For instance, in FIGS. 2A and 2B the « . j j- « -A. • * j u u-i*. 

uiuuu^v. Aiioiaiiv^., Ill JL jLvj^ inivi vii "casts a random die, with an associated probability p, of the 

neighbors of home router module 130fl include router mod- , . . j *u j . i * * *i. i i • 

1 i>!>A J u *u A' *i outcome 1, and sends the data packet to the lucky winner, 
ules 130o, 130c, and 130fl because they are directly con- 
nected to router module 130a via communication pathways easiest way to cast the random die is to generate a 
212 from one communication unit 210 of a neighboring uniformly random number between zero and one and deter- 
router module 130b-d to a communication unit 210 of the mine where this number falls in the cumulative probability 
home router module 130a. sequence {P,} given by Po=0, P,.=P,.i+p,- The random 

• uu _u J • •« n ^ 1 ™ •# f numbers can be generated "on the fly" or they can be stored 

The neighborhood supervisor unit 214 also monitors for . . ^ ^ ^ 

malfunctions and link failures in the data communication 25 * P'^'^^^- 

system. Along with monitoring, the neighborhood supervi- The second routing protocol, called a "Routing Wheel," 
sor unit 214 updates data in the dynamic load balancer 216 reduces the randomness of the Markovian protocol and is 
of the home router module 130 to be consistent with the easy to implement in hardware. Given the desired probabili- 
current physical state of the data communication system. ties p^, V2> • ■ Pk> for Aows to 1 through K neighboring 
The neighborhood supervisor unit 214 also reports detected 30 router modules 130 of a given router module, the routing 
network failures to a central fault reporting device located in table unit 218 of the given router module generates N (e.g,, 
the data communication system through a central fault 100) labels. The labels identify mdividual router modules 
monitoring function are assigned among the neighboring router modules 
Ihe dynamic load balancer unit 216 uses information ^^O according to the probability associated with eachneigh- 
coUected by the neighborhood supervisor unit 214 of the 35 ^'^^^S ^^^^^^ "'^^^^^^ ^""^ instance, the neighbonng router 
home router module 130 from the neighboring router mod- module 130"i" having probabihtyp, is assigned Np.numbcr 
ules 130. The dynamic load balancer unit 216 adjusts the labels identifymg that particular neighbonng router mod- 
routing tables of the routing table unit 218 based upon the ^1^- ^^^^ing table unit 218 then arranges the labels mto 
information coUected in order to optimize overaU utflization ^ conceptual distribution resembhng a physical circle where 
of the data communication system served by the network 40 ^^^^^ fyP.^^ ^redistributed fairly evenly around the 
traffic director system 110. The methods used by the conceptual circle. The routmg table unit 218 then sets a 
dynamic load balancer unit 216 will be discussed further conceptual pointer to a locaUon on the circle. When the 
below The routing table unit 218 provides information to ^y^^mic data flow splitter unit 218 services a data packet, it 
the dynamic data flow splitter unit 220 of the home router 1^°^ currently selected label on the cnrcle The 
module 130 in order that the data packets received by the 45 ^5^™'^ ^^^"^ ^P^"^' ^"'^ ^'''^ P^'^^^^ ^° 
communication unit 210 of the home router module 130 are neighboring router module 130 indicated by the current 
directed to destination networks 120 or router modules 130 label and then advances the pointer to the next label on the 
in a manner that preserves the integrity of the mathematical circle. The process never runs out of labels since when the 
models used by the dynamic load balancer unit 216. Pointer is at the label m position N, the pointer next advances 
^ , - J * « 1-4. * ™ *u ^« to the label in position 1. In further embodiments, a routing 
The dynamic data flow splitter umt 220 performs the 50 , , . ^ , ^ , • j 
■ 1 f J. 1 . ™' J u wheel is generated for each ongination/destmation pair and 
actual routing operation of data packets received by the ,-4 f 11 u- /n^T-m^ c\ Ar^/f^ e 
home router module 130 to direct fhe received data packets ^^^^^^y service level combination (OD/QoS). OD/QoS 
to other networks 120 or router modules. The division of combmations are discussed m detail below, 
labor between the routing table unit 218 and the dynamic Some higher level IP protocols are designed in such a way 
flow splitter unit 220 can be varied. In one embodiment, the 55 that makes the arrival of the data packets in an orderly 
routing table unit 218 prepares the routing data in a suitable sequential fashion at the final destination desirable. These 
form for the dynamic data flow splitter unit 220. The routing protocols can handle fallouts and out-of -sequence arrivals, 
table unit 218 then stores this routing data untfl the dynamic they become increasingly inefficient as the number of 
data flow splitter unit 220 requires the data. The dynamic out-of-sequence arrivals grows. In order to increase the 
data flow splitter unit 220 then looks up the routing data in 60 efficiency of such transfer protocols, the following modifi- 
the routing table unit 218 to determine how a particular data cation of flow splitting is used. This modification is incor- 
packet is to be routed. The dynamic data flow splitter unit porated into the third routing protocol and works with either 
220 in various embodiments of the present invention use Markovian splitting or Routing Wheel sphtting. 
different methods to determine how a received data packet For the third routing protocol, the granularity of splitting 
should be routed. Depending on which method is used, a 65 is changed for the appUcations running the higher level 
received data packet will be sent by either the communica- protocols mentioned above. Such protocols can usuaUy be 
tion unit 210 or the dynamic data flow sphtter 220 of the recognized from a device number in a packet header. The 
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dynamic flow splitter unit 220 is endowed with short-term communication system of which the network trafi&c director 

memory. When one ofthe "suspect" device numbers appears system 110 is part. A method used by the dynamic load 

in the header of a packet, the dynamic flow splitter unit 220 balancer 216 of the depicted embodiment used to accom- 

remembers the origin, the destination, the device number of plish this goal is described as follows, 
this packet, as well as the label of the neighboring router s As discussed above, each link 140 between one router 

module 130 to which the packet was sent. All subsequent module 130 and one network 120 and each link 150 between 

packets that have the same origin, destination, and device two router modules 130 is a data communication path that 

number that arrive while the memory persists are then sent allows for two-way traffic of data packets. Thus, each of 

to the same neighj?oring router module 130 as the original 140 and 150 of HG. 1 aUows data packet traffic to 
packet. In alternative embodiments, the short-term memory 10 travel in two directions either to or from a particular network 

is optionally refreshed by the later arrivals. 12O or router module 130. 

The fourth routing protocol used with the dynamic data Xo assist in describing the systems and methods below, 

flow splitter unit 220 relates to low latency and quality of each direction that a packet can travel along a link 140 or 

service. Latency of a system is the time required to send and 150 is referred to as an "arc." The term "arc'* does not refer 

receive information. Some applications, such as telephony, 15 particular physical part of a link 140 or 150 because 

video-telephony, or real-time interactive games or simula- the links allow packets to travel in either one of two 

tions require low latency levels. The latency requirements of directions. Rather, the term "arc" is conceptual since it refers 

other applications such as transmissions of email are not as to only one direction of travel in a link 140 or 150. Since 

stringent. The ordinary method for lowering latency for links 140 and 150 each aUow for two directions that packets 
urgent data packets is to give these packets a routing priority 20 ^.^j^ travel, each link 140 and 150 is assigned two conceptual 

while in routing queues as discussed below. arcs, one arc for each of the two directions of travel allowed 

Achieving additional speedups of urgent data packets is for within each link. Thus, each line of FIG. 1 which 
realized by making the dynamic data flow splitter unit 220 represents a link 140 or 150 could be replaced by two lines 
send more of the urgent packets over faster links and paths. representing two arcs which conceptually indicate the two 
The dynamic data flow splitter unit 220 then compensates directions of travel that are possible for packets to lake on 
for this by sending more of the ordinary packets over slower the link. The arcs of links 140 and 150 are iUustrated in FIG. 
links. The compensation can be adjusted to leave the overall 4 where each link has two arrows representing arcs for the 
performance of the routing system unchanged. In one two directions that packets can travel for any one link, 
embodiment, identification of the faster links by the network The depicted embodiment is not limited to links between 
traffic director system 110 in the data communication system "'^ networks 120 and router modules 130 but could include 
is readily accomplished because a high correlation exists links between other network devices or networks of any size 
between the highly used links and the faster links. This and topology. For example, links between routers and indi- 
condition is used to satisfy latency and quality of service vidual computers or terminals and links between large 
requirements by directing the urgent packets toward the networks such as the world-wide Internet and a country 
most frequented path choices for a given address group and sized corporate networks and small office local area net- 
by directing more of the less urgent packets away from the works are included in the depicted embodiment, 
most frequented path choices. Within router modules 130 there is an In node 420 and an 

The router module 130 also has a quality of service unit Out node 430 as shown in FIG. 4. Flow of the internal arcs 
310, a toll gate unit 312, an encoder-decoder unit 314, and 440 originate from the In nodes 420 and terminate at the Out 
a central fault monitoring unit 316 as shown in FIG. 3. The nodes 430 of each router module 130. The internal arcs 440 
quality of service unit 310 is concerned primarily with allow for additional parameters to be used in optimizing the 
latency regarding transmission of data packets in the data data communication system. To furnish additional 
communication system. The quality of service unit 310 is parameters, each In node 420 and Out node 430 is treated 
used to establish separate queues for applications with simUarly as how the router modules 130 are treated in the 
different latency and bandwidth needs. The queues are then various formulations for the depicted embodiments dis- 
served by appropriate frequencies of service and processing cussed below. 

rates. Also, billing rates are dependent upon the level of typical flow conditions in a data communication 

quality of service. The toll gate unit 312 collects the billing system, an overall flow in a particular arc typically is a 

statistics used to bill customers of the data communication conglomeration of one or more separate flows. Each separate 

system. These statistics primarily focus on billing. However, flow is characterized by its particular combination of flow 

they can be used to determine bandwidth demands and origination and flow destination expressed as its origination/ 

optimize routing network topology. destination (OD) pair. The originaUon and destination are 

To ensure privacy of communication, the data packets typically expressed at the network 120 or router module 130 
sent across network links that are vulnerable to eavesdrop- 55 level, so it is typically not expressed at a level specifying 

ping are encoded for transmission and decoded at the particular devices attached to a network. In some 

destination router module. The encoder-decoder unit 314 embodiments, the separate flows are characterized only by 

performs the encoding and decoding operations. This link OD pairs, so for these embodiments, the separate flows for 

security is not directly related to the routing operations of the a particular arc have unique OD pairs. Summing over these 

router module 130, Sufficient bandwidth for the encoder- unique OD pairs gives the overall flow f^^ for these arcs, 

decoder unit 314 is required so that transmission speed of the in the depicted embodiment, the separate flows are also 

overall router module 130 is not impeded. characterized by different quality of service (QoS) levels. In 

As described above, the dynamic load balancer unit 216 this embodiment, at least occasionally, a particular arc will 

iises information from the neighborhood supervisor unit 214 have more than one separate flow having the same OD pair, 
to determine parameters that the routing table unit 218 then 65 but having different QoS levels. To determine the overall 

uses to prepare routing table data. A goal of the dynamic load flow 4^ for these types of arcs, the separate flows would be 

balancer unit 216 is to optimize traffic flows in the data summed over both the OD pairs and QoS levels, giving 
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unique OD/QoS combinations. Characterization of the sepa- 
rate flows of an arc is not limited in the present invention to 
only OD/QoS combinations, but also incorporates other 
factors associated with traffic flow in a data communication 
system. For instance, in another embodiment, characteriza- 
tion of separate flows in an arc includes a device number of 
an origination or destination device attached to a network 
120. 

The overall flow of an arc is also constrained by an upper 
bound and a lower bound. The upper bound is typically 
proportional to the processing capacity of the arc for the 
depicted embodiment (e.g., 0.7 or 0.8 of the processing 
capacity of the arc). Each individual flow can also be 
constrained by its individual upper bound and individual 
lower bound. Again, the most common lower bound is zero 
for the depicted embodiment. 

The method to optimize traffic flow used by the dynamic 
load balancer unit 216 of the depicted embodiment defines 
an ideal theoretical state for data packet traffic in the data 
communication system as the case where there is zero flow 20 
throughout the data communication system while all 
demands or loads by users upon the data communication 
system are satisfied. A solution for data flows in the data 
communication system is' sought where the solution is a 
point in a solution space that has a minuBum distance to the 25 
theoretical ideal state. The solution is subject to certain 
physical constraints of a data communication system and is 
found through convex duality theory or convex optimization 
theory. 



would take on the form of the merit function. Optimization 
of a merit function would result in a maximization of the 
merit function. Other formulations that are equivalent to 
optimization of the penalty and merit funaions involve 
determining a point of equilibrium. As known in the art, 
once an optimization formulation is described, an equivalent 
formulation for a point of equilibrium is determined by 
using mathematical techniques known in the art. 

In a preferred embodiment, ry^fc=r^fc=l/c„fc; gy^i=g«&=0; 
and a and p are of the same magnitude. Then the solution 
of the optimization problem is the same as the solution of the 
competitive equihbrium problem using only the latter part of 
the equation, that is 



15 



over the non-supply/demand arcs. 

Other embodiments also include minimization of other 
penalty functions and maximization of other merit functions. 
Such penalty and merit functions are related to the depicted 
embodiment in that portions of these other penalty and merit 
functions are at least locally approximated by the expression 
Ea,p(/) above. These penalty and merit functions aU are 
advancements compared with the prior art, since they are 
used to optimize flows in a stochastic envirorunent of a data 
communication system. 



The expression E o(f) incorporates factors associated 
In the depicted embodiment, the solution for data flows in 30 ^ith individual OD/QoS combinations for each arc "ab'* 



a data communication system is uniquely formulated to 
optimize a variety of network objectives and conditions 
compared with prior art formulations that focus on one 
known data rate objective or condition subject to conserva- 
tion of data flows. Data flows are conserved when the 
amount of flow coming into a node (typically a router 
module 130) equals the amount of flow leaving the node. 
Any data either produced or destroyed at a node is factored 



35 



over all the arcs in a data communication system. Each 
individual OD/QoS combination "j" of an arc "ab" has a 
particular resistance parameter, Zj^j, related to any hin- 
drances to flow and a goal parameter g^^^ related to the 
current demand or load by users. 

For the depicted embodiment, the resistance parameter for 
an individual arc "ab" and OD/QoS combination "j" is also 
uniquely formulated, being proportional to the reciprocal of 
the overall data flow capacity, Cy^^ of the individual arc "ab" 



into data flow conservation. 

The solution for data flows also optimizes the foUowing 40 and OD/Qo,S combination "j".'TnEius, r^ 
uniquely formulated expression E^ ^{f) involving a substan ' ' ' 

tially quadratic function of data flows in a data communi- 
cation system: 



.ab yCj,aby where 



^//tb maximum amount of data flow which 



the 



45 



individual arc "ab" is capable of supporting for OD/QoS 
combination "j". For instance, C^fr^Cy^^ and tab-^}\ab 
for all j. In component flows, most commonly 



50 



where the subscript "j" indicates a particular OD/QoS com- 
bination and the subscript "ab" indicates a particular arc 
"ab". The term f^^^ is for flow associated with a particular 
OD/QoS combination "j" and a particular arc "ab". The 
terms r^-^ and gj^^ are resistance and goal parameters, 
respectively, for the particular OD/QoS combination "j" and 
arc "ab". The multiplication factors a and p are used to tailor 
emphasis between the two component expressions for 
E„ p(/). Since only the relative magnitude of a and p are 
important, wc implicitly assume the quantities are scaled to 
satisfy a+p=l. 

In the form above, the expression E„ p(f) is a penalty 
function which represents undesirable influences and results 
affecting communication flow in a data communication 
system as have been described. Optimization of the expres- 
sion E^ p(j^ as a penalty function would result in minimi- 
zation of E„ p(j). If the expression E„^(/) was negated it 
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The resistance parameter is also used to designate that an arc 
for an OD/QoS combination is not functional by setting r^^^^, 
to a relatively large number. The solution to the optimization 
of the uniquely formulated expression E^^ p(j) over aU the 
component flows /^^t of each OD/QoS combination "j" for 
each arc "ab" over all arcs and OD/QoS combinations of a 
data communication system results in solutions of flow Sj^^^, 
for each OD/QoS combination "j" for each arc "ab" in the 
data communication system. 

Methods that could be used to solve the optimization of 
the uniquely formulated expression E„ p(i) include methods 
that have been used to solve other optimization formulations 
such as a projected gradient method, and a hfll climbing 
method. The depicted embodiment uses a unique distributed 
optimization method incorporated in the dynamic load bal- 
ancer unit 216 for optimizing the expression E(^p(/). This 
method is uniquely formulated for stochastic network flow 
that is uncertain, rapidly changing and random and topology 
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that can stochastically change. The distributed optimization 
method is generally easier to implement and converges more 
quickly to a solution compared with prior art methods that 
have been used on other formulations. The distributed 
optimization method also incorporates conservation of flows 
and unique formulations of gy and Xj^i,. 

As part of this distributed optimization method, the 
depicted embodiment defines a set of quantities at each 
router module 130 that are related to flows. Each quantity is 
referred to as a "potential." The set of potentials for a 
particular router module 130 has one potential for each 
possible OD/QoS combination knovra to the particular 
router module 130. Possible OD pairs are determined by the 
various combinations of a data packet origination and a data 
packet destination. These combinations would not typically 
include combinations where both the origination and desti- 
nation are the same. TypicaUy, the data packet origination 
and destinations would be networks 120 and router modules 
130, but could include other network devices at a more 
discrete level as discussed above, such as individual com- 
puters or terminals. Thus, the possible OD pairs of the 
OD/QoS combinations for a particular router module 130 
would typicaUy include the OD pairs derived from all data 
packet originations and data packet destinations known to 
the particular router module 130. 

Typically networks 120 and router modules 130 are at 
times data packet originations and at other times data packet 
destinations. As such, the same two components of a first 
OD pair could be the components of a second OD pair by 
designating the first pair origination as the second pair 
destination and the first pair destination as the second pair 
origination. For small sized data communication systems, a 
router module 130 stores one potential for each possible 
OD/QoS combination for the entire data communication 
system. For a larger sized data communication system, an 
aggregation scheme is used which reduces the number of 
potentials stored by each router module 130. 

An aggregation scheme reduces the number of networks 
120 and router modules 130 known to any one router module 
130 in order to keep the set of potentials stored in the router 
module to a manageable size. As shown in an aggregation 
hierarchy 510 of FIG. 5, this aggregation scheme has a 
physical portion 512 with physical nodes and a conceptual 
portion 514 with conceptual nodes. The physical nodes 
include networks 120 and router modules 130. The concep- 
tual nodes each serve as labels for a particular collection 
made up of any number of physical nodes and/or other 
conceptual nodes. The physical nodes of the physical portion 
512 are typicaUy found in a particular geographical region. 
FIG. 5 only illustrates a representative example, and the 
depicted embodiment can have other regions of various sizes 
and locations. 

Although no. 5 illustrates an aggregation hierarchy in 
terms of geography, the depicted embodiment is not limited 
to only hierarchies defined in geographical terms. Other 
hierarchies defined in terms of an organizational, political, 
topical, topological, or other types of structures are also 
included in the present invention. Also, FIG. 5 is not 
intended to limit the present invention to any particular 
manner or type of geography in defining the physical portion 
512 of the aggregation hierarchy 510. The physical nodes 
can be grouped according to other common geographical 
regions besides the manner of using cities as regions as 
illustrated in FIG. 5. 

For iUustration purposes, FIG. 5 shows the regions of 
Seattle, Portland, San Francisco (S.F.), Los Angeles (L.A.), 
Boston, New York (N.Y), Atlanta, and Miami having physi- 
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cal nodes 520-534 re^ectively. The Seattle region and 
Portland region represented by conceptual nodes 540 and 
542 are further aggregated into the Northwest (N.W.) region 
represented by conceptual node 560. Similarly, conceptual 

5 nodes 544 and 546 for the S.F. and L.A. regions respectively 
are further aggregated into the southwest (S.W.) region 
represented by concepmal node 562. Conceptual nodes S4S 
and 550 for the Boston and N.Y. regions respectively are 
aggregated into the Northeast (N.E.) region represented by 

10. conceptual node 564. Conceptual nodes 552 and 554 for the 
Atlanta and Miami regions respectively are aggregated into 
the Southeast (S.E.) region represented by conceptual node 
566. The conceptual nodes 560 and 562 for the N.W. and 
S.W. regions respectively are aggregated into the West Coast 

15 region represented by conceptual node 570. The conceptual 
nodes 564 and 566 for the N.E. and S.E. regions respectively 
are aggregated into the East Coast region represented by 
conceptual node 572. The conceptual nodes 570 and 572 for 
the West Coast and East Coast regions respectively are 

20 aggregated into the USA region represented by the concep- 
tual node 574. 

Physical nodes of the various regions in the physical 
portion 512 are directly connected with other physical nodes 
(typically router modules 130) in other regions of the 

25 physical portion. For instance, a physical node 520c in the 
Seattle region is directly connected to a physical node S22a 
in the Portland region through link 580. The various regions 
are connected together via this type of direct connection 
between physical nodes in one region and other physical 

30 nodes (typically router modules 130) in another region. Not 
all physical nodes in one region need to be connected to 
another physical node in another region. This is because 
physical nodes within the same region are connected to one 
another either directly or through other physical nodes 

35 (typically router modules 130) in the same region. Subnet- 
works formed by the aggregation scheme need to maintain 
connectivity. Thus, between each pair of nodes in a subnet- 
work there must be a connection. In the aggregation scheme 
illustrated in FIG. 5, the physical nodes 520fl--c in the area 

40 of Seattle afl share a common conceptual node 540 in the 
conceptual portion 514 labeled Seattle. Each physical node 
520a-c in the area of Seattle stores one potential for each 
possible OD/QoS combination based on the physical nodes 
520 composed of networks 120 and router modules 130 in 

45 physical area 512 associated with the Seattle conceptual 
node 540. Also, under the same aggregation scheme, each 
physical node 520a-c in the Seattle area further stores one 
potential for each possible OD/QoS combination based also 
on the conceptual nodes that are blackened in on FIG. 5. The 

50 resultant OD/QoS combinations include any OD pair com- 
bination based on either physical nodes, conceptual nodes or 
a mixture of both physical and conceptual nodes as the 
originations and destinations of the OD pairs. The concep- 
tual nodes that are blackened in on FIG. 5 are SeatUe 

55 conceptual node 540, Portland conceptual node 542, N.W. 
conceptual node 560, S.W. conceptual node 564, West Coast 
conceptual node 570, East Coast conceptual node 572, and 
USA conceptual node 574. As shown in FIG. 5, the physical 
node 520c is directly connected to physical node 522fl via 

60 physical link 580 and physical node 524fl via physical hnk 
582. Due to these direct connections, physical node 520c 
also stores potentials for OD/QoS combinations that incor- 
porate one or both of physical nodes 522fl and 524a as either 
origination, destination or both origination and destination. 

65 Thus, physical node 520c stores more potentials due to more 
OD pairs than the other physical nodes 520. In an alternate 
embodiment only physical nodes in the same region as the 
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node in question along with appropriate conceptual nodes 
are used for possible OD pair combinations. For instance in 
the alternative embodiment, physical node 520c only stores 
potentials for physical nodes 520a-c in physical portion 512 
and the blackened conceptual nodes. In the alternative 
embodiment, physical node 520c does not store potentials 
for any other directly connected nodes such as physical 
nodes S22a and 524^. 

There are general rules to designate which physical and 
conceptual nodes are used to derive OD pairs for the 
OD/QoS combinations for a particular physical node. First, 
a router module 130 (at physical node 520c in the example 
of FIG. 5) stores potentials for each OD/QoS combination 
having an OD pair based on the physical nodes (520b and 
520c in the example of FIG. 5) witti which the router module 
130 shares a common conceptual node (540 in the example 
of FIG. 5). Second, if the router module (at physical node 
520c in the example of FIG. 5) is directly connected to other 
physical nodes (522fl and 524fl in the example of FIG. 5), 
the other physical nodes are also used to derive other OD 
pairs for OD/QoS combinations for which potentials are 
stored in the router module (at physical node 520c). Finally, 
the router module (at physical node 520c) will store poten- 
tials for OD/QoS combinations for OD pairs that incorporate 
one or both of conceptual nodes that are either "lineal 
ancestors" of the router module (at physical node 520c) or 
"siblings" of the lineal ancestors. For the example of FIG. 5, 
the conceptual nodes that are Hneal ancestors of the physical 
nodes 520 (including router modules 130) are Seattle con- 
ceptual node 540, N.W. conceptual node 560, West Coast 
conceptual node 570 and USA conceptual node 574. The 
conceptual nodes in the example of FIG. 5 that are siblings 
of the Uneal ancestors are Portland conceptual node 542 
(sibling of Seattle conceptual node 540), S.W. conceptual 
node 562 (sibling of N.W. conceptual node 560), and East 
Coast conceptual node 572 (sibling of West Coast concep- 
tual node 570). 

The method for optimizing network flow in a data com- 
munication system used by the depicted embodiment of the 
dynamic load balancer unit 216 is further based on a 
difference function The difference function 

^Cf/>fc?P;>i) is evaluated for each arc "ab" for each OD/QoS 
combination "j" for each router module 130 in the data 
communication system. When the difference function 
dCf/>A»Pj>fc) is evaluated for a particular router module 130, 
that router module is designated a home router module. The 
difference function d(/^- „fr,p/^fc) is evaluated for each arc that 
either originates from or terminates with the home router 
module. Router modules 130 that are physically connected 
to the home router module are designated neighboring router 
modules. The neighboring router modules share with the 
home router module 130 the arcs either originating or 
terminating with the home router module potentials. Poten- 
tials py and py^ are at the origination and destination router 
modules 130 respectively of arc ab for OD/OoS combination 
"j". Thus, the home router module 130 and neighboring 
router modules 130 are either origination or destination 
router modules, depending upon the origination and desti- 
nation of the particular arcs being evaluated. 

For a particular arc "ab" involved in the evaluation, the 
difference function d(j'y^fc,py^£,) is evaluated for each com- 
mon OD/QoS combination "j" that is known both to the 
home router module 130 and the neighboring router module 
130 of the particular arc "ab." The resultant expression for 
the difference function d{fj,,i,,pj„h) for arc "ab" and 
OD/QoS combination "j" is 
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where 



Sab - ^gj.d> ~ ^ ~ X 

s I **> J 

The potentials py ^ and Py ^ are potentials for router modules 
130 "a" and "b" 'respectively for the "j" OD/QoS combina- 
tion. Flows /jt^ where kj-j are flows for the arc "ab" 
associated with OD/QoS combinations other than OD/QoS 
combination "j". 

The first term, (ry^i/2)*(g,i,-/y^^)^ of the difference 
function d(ff^b*Vj,ab) is a simplified version of the expres- 
sion for p(/y^^) where only one arc "ab" and one 
""^ OD/QoS combination "j" is considered. The difference 
function d(Jy^^,py ^^) is subject to the following boundary 
conditions: 

20 

where 



25 



30 




and where ly^^ and 1^^, are lower limits, and Uy^^ and u^^ are 
upper hmits on fj^i, and respectively. 

35 An intuitive explanation of the difference function 
difj,ab>Pf.ab) is that the first term related to E^p(/y^^fc) rep- 
resents undesirable results and conditions of data packet 
traffic being transmitted over a data communication system, 
such as delays, poor utilization of resources, poor adaptabil- 

40 ity to sudden changes, etc. For a given arc, when the goal 
parameter, g^^, equals zero and the resistance parameter 
equals the inverse of capacity of the arc, then the first term 
of the difference function d(/y ,,£„py^^) equals the number of 
jobs per unit time that have to go into a waiting queue 

45 because a router module 130 is busy. The second term 
fj\ab*(pj,a-^j,b} represents effective utilization of the data 
communication system as subject to constraints imposed by 
the data communication system such as flow constraints. 
The difference function d(/y^^,py^^) thus represents a dif- 

50 ference between desirable and undesirable aspects involved 
with the data communication system. 

As part of finding a local equilibrium point, by differen- 
tiating the difference function d(fj^ab7pj,ab) ^^^ch arc 
between router modules with respect to the flow fj of the 

ss OD/QoS combination "j" and arc "ab", setting the resultant 
differential equal to zero, and solving for the flow, jy^^, a 
response function results for flow fj^ab- ^ iUustrated in FIG. 
6, for a particular arc "ab" 610 having beginning router 
module "a" 620 and ending router module "b" 630 and for 

60 a common OD/QoS combination "j", the response function 
of the optimal flow fy^^^^ 640 is: 

/y^t (p)=max{l;..i,. min {\,i„^tr^j^-Vi^)lh^b}} 

where min and max are minimum and maximum functions 
65 which choose the minimum or maximum term of the pair of 
terms that arc bracketed. For instance, max (a,b) chooses the 
maximimi value between a and b. Thus, max (a, min (b,c)) 
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first chooses the minimum between b and c and then chooses 
the maximum between a and the minimum choice between 
b and c. Minimum flow for J^^^ is typically zero. Maximum 
flow for fj^i, is typically a percentage of arc capacity. 

The goal parameter, g^^, and resistance parameter, r^^^, 
are for the particular arc "ab" and OD/QoS combination "j". 
The potential p^^ is for the arc beginning router module "a" 
620 and the potential p^^ is for the arc ending router module 
"b" 630. Both potentials py^ and py^ are associated with a 
common OD/QoS combination "j". Typically, the origina- 
tion and destination nodes for an OD/QoS combination are 
not the arc beginning and arc ending router modules 620 and 
630 respectively. Also, the origination and destination nodes 
for an OD/OoS combination can be a combination of 
physical and/or conceptual nodes whereas the arc beginning 
and arc ending nodes are always physical since they are 
router modules 130. 

Summing these optimal flows for arc beginning "a" and 
arc ending "b" router modules 620 and 630 for arc "ab" over 
the pD/QoS combinations that are known to the arc begin- 
ning and ending router modules results in the optimal flow 
for arc "ab". This response function of flow describes an 
optimized flow based on certain values for potentials. These 
potential values further satisfy a constraint based on how 
data packets are created and terminated in the data commu- 
nication system. 

In the depicted embodiment, the potentials are indepen- 
dent variables and the flows are dependent variables. 
However, in other embodiments, different subsets of flows 
and potentials are used so that some flows and potentials are 
dependent variables and other flows and potentials are 
independent variables. 

The depicted embodiment of the invention is configured 
so that all data packet traffic originates and terminates at 
network terminals of network 120 or router modules 130. 
This means that all data packet traf&c having a common 
destination, either created at a particular router module 130 
or arriving at the particular router module, must equal the 
data packet traffic with the common destination leaving the 
router module 130. This requirement is referred to as a 
conservation of flows or a flow constraint. When imple- 
menting the method of the depicted embodiment, the router 
modules 130 scattered throughout a data communication 
system frequently adjust the data packet routes in the data 
communication system to pursue the goal of maximizing net 
benefits for stochastic distributions of network loads and 
stochastic changes in topology. The adjustments arc done by 
manipulating potential values according to the conservation 
of flows. 

Adjustments are first needed when a data communication 
system 110 is initially brought into service. Adjustments are 
subsequently needed due to the continually changing 
demands or loads by users of the data communication 
system and changes in the topology of the data communi- 
cation system due to growth of the data communication 
system or equipment failures within the existing data com- 
munication system. The method of the depicted embodiment 
incorporated into the dynamic load balancer units 216 of the 
router modules 130 is responsible for these adjustments 
made by the router modules. 

As stated above, the dynamic load balancer unit 216 uses 
information collected by the neighborhood supervisor unit 
214. This collected information can now be described in 
terms of the response function for flow, /y^^, as discussed 
above. The discussion will be focused on a particular router 
module 130 designated the "home" router module, however, 
the home router module discussed is representative of all 
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router modules in that aU router modules also perform the 
steps described. 

In general, there are two main approaches that are used in 
the present embodiment to determine the flows for a given 

S home router module 130. In both approaches, arcs are 
distinguished as being either supply arcs or non-supply arcs, 
A supply arc either directly connects an origination network 
120 or origination user device of a network for a given 
OD/QoS combination to the given home router module 130 

10 or a router module between the network or user device and 
the home router mode where there is no branching of the 
arcs for the home router module and the network or user 
device. Non-supply arcs are also directly connected to the 
given home router module but are not directly connected to 

15 an origination node. 

For both approaches, the goal parameters for a supply arc 
for a given OD/QoS combination are set equal to the user 
demand or load placed on the supply arc. For instance, if 
user demand or load was 100 Mbits/sec on a supply arc for 

20 a given OD/QoS combination then the goal parameter for 
the same supply arc for the same OD/QoS combination 
woidd be set to 100 Mbits/sec. The neighborhood supervisor 
unit 214 senses what the demand or load is from any 
networks 120 on network devices that are directly connected 

25 to the home router module 130. Also, for both approaches 
the resistance parameter for a given OD/QoS combination 
would be set to a relatively large number for a supply arc. 
The large number means that there is a high cost incurred if 
the goal of the goal parameter (in this case 100 Mbits/sec) 

30 is not met. For a non-supply arc the resistance parameter for 
a given OD/QoS combination is set to the inverse of the arc 
capacity for the given OD/QoS combination. 

The two approaches to solve for flows for a given router 
module differ in how they deal with the goal parameters for 

35 non-supply arcs. The first approach solves for flow for one 
OD/QoS combination at a time whereas the second approach 
solves for flow for all OD/OoS combinations at once. For the 
first approach, the goal parameter for a non-supply arc is set 
to the negative of the historical flow for an OD/OoS com- 

40 bination for the arc. The historical flow for an arc is the most 
recent flow amount recorded. Often times historical flows 
tend to change more slowly than demand or load, especially 
for arcs that have become a major artery for a data com- 
munication system. For instance, if the flow on a non-supply 

45 arc was most recently recorded as 50 Mbits/sec for an 
OD/QoS combination, then the goal parameter for that arc 
would be -50Mbits/sec. 

With the first approach, for a given home router module 
130, the dynamic data flow splitter unit 220 determines 

50 historical flows from header information on data packets 
sent from the home router module. These historical flows of 
the home router module for sent packets are stored in the 
dynamic data flow sphtter unit 220. The historical flows are 
represented by averages, variances and other expressions 

55 known in the art of statistics and mathematics. The neigh- 
borhood supervisor unit 214 queries the dynamic data flow 
splitter unit 220 for the historical flows of sent data packets 
as needed for flow calculations or to send to other neigh- 
boring router modules. The historical flows of sent data 

dp packets are for arcs directly connected to the given home 
router module 130 handling packets sent by the home router 
module to networks or other router modules. In the depicted 
embodiment, the neighborhood supervisor unit 214 of the 
given home router module 130 also receives updated his- 

65 torical flow information from other neighborhood supervi- 
sors for arcs that arc directly connected to the given home 
router module and carry data packets to the given home 
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router module. A special case of the first approach is where 
all historical flows are equal to zero. This special case occurs 
at such times as the initial start-up phase of a data commu- 
nication system. 

The second approach to solve for flows for a given home s 
router module 130 uses computed flows instead of historical 
flows. Flows are determined for all OD/QoS combinations at 
once through an iterative process. For a K'^ iteration, the 
overall flow for an arc is determined by summing all flows 
for the OD/QoS combinations for the arc solved during the lo 
K''' iteration. The goal parameter for the non-supply arcs for 
the K+1''' iteration are then equal to the negative of the 
overall flows for the arcs for the K'* iteration. 

The techniques used to solve in this iterative fashion the 
flows for a home router module 130 for all OD/QoS com- is 
binations at once require extensive computing capability- 
Some embodiments use the second approach to solve for 
flows for all OD/QoS combinations at once. The depicted 
embodiment as incorporated into the methods of the 
described neighborhood supervisor unit 214 uses the first 20 
approach to solve for flows one OD/QoS combination at a 
time for a given home router module starting with Step 710 
of FIG. 7. 

With the first approach, the neighborhood supervisor unit 
214 of the home router module 130 collects potentials and 25 
historical flows from all neighboring router modules 130 
directly connected to the home router module in step 720 of 
the flowchart of FIG, 7 showing the method used by the 
depicted embodiment. The coUected potentials and histori- 
cal flows are for common OD/QoS combinations known to 30 
both the home router module and at least one neighboring 
router module. Either the neighboring router modules or the 
home router module initiates transmission and reception of 
the potentials and historical flows collected from the neigh- 
boring router modules to the home router module. In the 35 
depicted embodiment, the neighboring router modules ini- 
tiate the transmission of their potentials and historical flows 
to the home router module. However, in other embodiments, 
the home router module initiates the transmission. The 
dynamic load balancer unit 216 of the home router module 40 
130 then uses the collected potentials and historical flows 
along with the potentials and historical flows of the home 
router module to calculate, via the flow response function, 
the data packet flows for each OD/QoS combination to and 
from the home router module for all arcs connected to the 45 
home router module. For each connection between the home 
router module 130 and neighboring router modules or net- 
works 120, there are two arcs as described above. One arc 
has the home router module as the origination node and the 
other arc as the destination node. In other embodiments, 50 
only one-sided connections are possible so that for each 
connection there is only one arc. 

The neighborhood supervisor unit 214 then uses the 
historical flows and demand or load flows to update the goal 
parameters for afl the non-supply and supply arcs respec- 55 
lively of the home router module 130 in step 722 of FIG. 7. 
TypicaUy, the goal parameters for non-supply arcs are set lo 
the negative of the historical flow for a given OD/QoS 
combination and the goal parameters for the supply arcs are 
set to the demand or load flow on the arc. The neighborhood 60 
supervisor unit 214 also identifies any arcs that have zero 
overall flow based on their historical or demand/load flow so 
that any subsequent modifications of the home potentials are 
done with the zero flow conditions in mind as explained 
further below. 65 

InitiaUy, no OD/QoS combinations are selected by the 
neighborhood supervisor unit 214, so the outcome of deci- 



sion 724 of FIG. 7 regarding whether all OD/QoS combi- 
nations have been selected is "no." In such case, the neigh- 
borhood supervisor unit 214 selects an OD/QoS 
combination in step 726 that has not been chosen since the 
last time the start step 710 was performed. For all arcs either 
originating from or terminating into the home router 
module, the neighborhood supervisor unit 214 of the home 
router module passes, in step 728, all the potentials of the 
home router module and those collected from neighboring 
router modules for the selected OD/OoS combination to the 
dynamic load balancer unit 216. The neighborhood super- 
visor unit 214 also passes all the goal parameters and 
resistance parameters for the arcs of the particular router 
module 130 based on the most recent historical and demand/ 
load flows collected by the neighborhood supervisor unit 
214 to the dynamic load balancer unit 216. . 

The dynamic load balancer unit 216 then determines, for 
the selected OD/QoS combination, the flows across the arcs 
of the home router module. The dynamic load balancer unit 
216 then determines, in decision step 730 of FIG. 7 whether 
the determined flows are conserved for the selected OD/QoS 
combination or whether the time period for iteration has 
expired. Flows are conserved when the incoming flows plus 
flows originated by the home router module equals the flows 
leaving the home router module. If flows are not conserved 
for the selected OD/QoS combination or the time for itera- 
tion has not expired, the decision step in 730 is "no" and the 
method branches to step 732 where the dynamic load 
balancer mdi 216 adjusts the potentials for the home router 
module 130 for the selected OD/QoS combination. After the 
home potentials are adjusted in step 732, flow conservation 
is again checked in decision step 730. The home potentials 
are iteratively adjusted through the combination of potential 
adjustment in step 732 and flow conservation verification in 
decision 730 until flow is conserved and decision step 730 
is "yes" and the method branches back to step 722 to update 
the goal parameters. Iteration techniques to allow for rapid 
convergence to solutions involving iteration are known in 
the art and include over or under-relaxation techniques. In 
one embodiment, the, potentials are adjusted one at a time by 
the iteration process for a given OD/QoS combination until 
flow is conserved. In another embodiment all the potentials 
for a router module for all the OD/QoS combinations 
associated with the router module are adjusted for each 
iteration of a Gaussian or other second order method of 
determining solutions by block techniques known in the art. 

If the neighborhood supervisor unit 214 has identified any 
zero flow arcs that have zero overall flow when the goal 
parameters are initially updated in step 722 immediately 
after the historical and demand flows are obtained in step 
720, then during the home potential adjustment in step 732, 
a goal parameter associated with a zero flow arc can be set 
to the negative of any calculated flow for the zero flow arc 
resxdting from adjustments of potentials for an OD/QoS 
combination to ensure that the flow for the zero flow arc 
remains small. The new value for the goal parameter for the 
zero flow arc is then used to update the goal parameters in 
step 722. At the completion of adjusting potentials for all 
OD/QoS combinations, the updated goal parameter for a 
zero flow arc will then be the summation of flows over all 
OD/QoS combinations for the particular zero flow arc due to 
adjustment of potentials. 

Once any goal parameters are updated in step 722 another 
OD/QoS combination is selected which had not been pre- 
viously selected since the last performance of start step 710 
and steps 728-732 are again performed for the newly 
selected OD/QoS combination. After steps 728-732 are 
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completed, again any changed goal parameters are updated. 
The process similarly cycles through the OD/QoS combi- 
nations until decision step 724 determines that all OD/QoS 
combination have been selected, whereby the "yes" branch 
is taken and decision step 734 determines if any potentials 
of the home router module 130 had been adjusted by step 
732, 

If any home potentials were adjusted, then decision step 
734 branches to step 736 of FIG. 7 where the neighborhood 
supervisor unit 214 of the home router module 130 reports 
the adjusted home potentials and historical flows to the 
neighboring router modules. Decision step 738 then deter- 
mines if the period of time since the neighborhood super- 
visor unit 214 of the home router module 130 last reported 
home potentials to neighboring router modules is greater 
than a predetermined amount of time, T^. If the period is not 
greater then Tr, decision step 738 branches back to the start 
step 710. If the period is greater than T^, decision step 738 
branches to decision step 740, the same step to which 
decision step 734 branches if no home potentials were 
adjusted. In decision step 740 a determination is made as to 
whether any adjusted home potentials have changed more 
than a predetermined threshold Jj- since the last time elec- 
tronic tables of the dynamic load balancer unit 216 storing 
the home potentials have been updated with adjusted home 
potentials. If any changes in the adjusted home potentials are 
greater than Tj> decision step 740 branches to step 742 
wherein the electronic tables are updated with the latest 
values for the home potentials. The dynamic load balancer 
unit 216 subsequently updates the routing table unit 218. If 
there are no changes in the adjusted home potentials greater 
than Tj, decision step 740 branches to the start step 710. 

The present invention is not limited to the particular 
routines used by the depicted embodiment to report updated 
potentials to neighboring router modules 130 or in updating 
electronic tables storing home potentials. Other embodi- 
ments use other methods, intervals, or thresholds to deter- 
mine when to report to neighboring router modules 130 and 
when to update electronic tables storing home potentials. 

In an alternative embodiment, the resistance parameter for 
each particular arc is a function of the flows of each OD/QoS 
combination. For instance, for a particular OD/QoS 
combination, the resistance, r, is as follows: 



iffic-f)) 



where f is the flow for an arc for an OD/QoS combination 
and c is the capacity or residual capacity of the arc. It follows 
that since the response function expresses the dependency of 
flow for an OD/QoS combination for an arc, the resistance 
parameters are also expressible in terms of potentials. 

From the foregoing it will be appreciated that, although 
specific embodiments of the invention have been described 
herein for purposes of illustration, various modifications 
may be made without deviating from the spirit and scope of 
the invention. Accordingly, the invention is not limited 
except as by the appended claims. 

What is claimed is: 

1. In a data communication system having network traffic 
with randomly distributed demands and flows, a network 
traffic director system comprising: 

router modules configured to direct data packets in the 
data communication system, each router module being 
a home router module comprising: 
a neighborhood supervisor configured to send home 
potentials of the home router module to neighbor- 



hood supervisors of neighboring router modules and 
to receive neighbor potentials of the neighboring 
router modules from neighborhood supervisors of 
neighboring router modules; 

5 a dynamic load balancer configured to determine flows 
based on the home and neighbor potentials, to adjiist 
the home potentials if first conditions including flow 
conditions are not met, and to update routing tables 
if second conditions based on the adjusted home 

10 potentials are met; and 

a dynamic data flow splitter configured to receive data 
packets from networks and router modules, to select 
a portion of the data communication system for each 
data packet received based on the updated routing 

15 tables wherein each received data packet is trans- 

mitted to the portion of the data communication 
system selected by the dynamic data flow splitter for 
the received data packet. 

2. The network traffic director system of claim 1 wherein 
20 each of the home potentials of each home router module is 

associated with a pair of nodes of the data communication 
system comprising an origination node and a destination 
node. 

3. The network traffic director system of claim 1 wherein 
25 each of the home potentials of each home router module is 

associated with a quality of service level. 

4. The network traffic director system of claim 1 wherein 
the neighborhood supervisor is configured to send the home 
potentials to neighborhood supervisors of neighboring 

30 router modules when the flow conditions are not met. 

5. The network traffic director system of claim 1 wherein 
the flow conditions comprise conservation of flows. 

6. The network traffic director system of claim 1 wherein 
the dynamic load balancer is configured to determine flows 

35 by using a response function associated with optimizing at 
least one of a penalty function or a merit function for 
stochastic demands and flows of the data communication 
system. 

7. The network traffic director system of claim 1 wherein 
40 the dynamic data flow splitter is configured to select the 

portion of the data communication system for each data 
packet received further based on a Markov protocol. 

8. The network traffic director system of claim 1 wherein 
the dynamic data flow splitter is configured to select the 
portion of the data communication system for each data 
packet received further based on a Routing Wheel protocol. 

9. The network traffic director system of claim 1 wherein 
the dynamic data flow splitter is further configured to select 
a portion of the data communication system based on 
previously selected portions of the data Communication 
system when an address for a received data packet is a same 
type as an address stored in the dynamic data flow splitter. 

10. The network traffic director system of claim 1 wherein 
the dynamic data flow splitter is further configured to select 
of a portion of the data communication system based on a 
quahty of service level associated with a data packet. 

11. In a communication system having, network traffic 
with flows, a network traffic director system comprising: 

router modules, each router module being a home router 
60 module configured to store and adjust home potentials 
of the home router module, and configured to receive 
and store neighbor potentials of neighboring router 
modules, the home router modules configured to deter- 
mine ideal data flows using the home and neighbor 
65 potentials with an optimization of at least one of a merit 
function or a penalty function involving stochastic 
changes in at least one of demands or topology in the 
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communicatioD system, the home router modules con- 
figured to receive and route network traffic based on the 
home and neighbor potentials, 

12. The network traffic director system of claim 11 
wherein home router modules are configured to be a neigh- 5 
bor router module of other home router modules. 

13. The network traffic director system of claim 11 
wherein the router modules arc configured to continue 
adjusting home values to seek flow conservation until flow 
conservation is satisfied or until a time period has expired. lO 

14. The network traffic director system of claim 11 
wherein the merit or penalty function are at least locally 
approximated by a quadratic function of the flows. 

15. The network traffic director system of claim 11 
wherein the neighbor potentials are stored according to an 15 
aggregation scheme. 

16. The network traffic director system of claim 11 
wherein the ideal data flows are further based on resistance 
of arcs associated with the router module wherein each 
resistance of an arc is configured to be a function of arc 20 
capacity. 

17. The network traffic director system of claim 11 
wherein the ideal data flows are further based on resistance 
of arcs associated with the router module wherein each 
resistance of an arc is configured to be a function of flows 25 
of the arc. 

18. The network traffic director system of claim 11 
wherein the neighbor router modules are connected to the 
home router modules without any other router modules 
therebetween. 

19. The network traffic director system of claim 11 
wherein the neighbor router modules are configured to be 
associated with the home router module through at least one 
of a political, organizational, topological, topical, or geo- 
graphical relation. 

20. The network traffic director system of claim 11 
wherein each router module is configured to adjust home 
potentials to an equilibrium point for a difference function 
associated with flows of arcs associated with the router 
module, the difference function associated with differences 
between home potentials and neighbor potentials. 

21. The network traffic director system of claim 11 
wherein each router module stores adjusted home values 
when adjustment of home values is greater than a threshold. 

22. The network traffic director system of claim 11 
wherein each home router module transmits adjusted home 
values to its neighbor router modules based on how much 
time has elapsed since last transmitting adjusted home 
values. 

23. The network traffic director system of claim 11 so 
wherein the merit function or penalty function involves costs 

of congestion comprising at least one of trafBc delays, high 
latency, diminished throughput, lost packets, or unrespon- 
siveness to sudden changes in topology or loading. 

24. The network traffic director system of claim 11 55 
wherein the neighbor router modules are connected to the 
home router modules wherein each connection has only one 
arc. 
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25. In a data communication system having network 
traffic, a network traffic director method comprising: 

storing home potentials associated with a home router 
module; 

receiving and storing neighbor potentials associated with 
neighbor router modules of the home router modxile; 

determining ideal flows based on the home and neighbor 
potentials and optimizing at least one of a merit func- 
tion or a penalty function for random changes in 
demands or topology in the data communication sys- 
tem; 

determining conservation of flows for the home router 
module; 

adjusting the home potentials to approach a slate where 
flows arc conserved if determined not to be conserved; 

reporting adjusted home potentials to the neighbor router 
modules; and 

routing network traffic based on home and neighbor 
values. 

26. The network traffic director method of claim 25 
wherein the routing further includes using at least one of a 
Markov protocol, a Routing Wheel protocol, stored data 
packet addresses, or a level of quality of service of received 
communication traffic to be routed. 

27. The network traffic director method of claim 25 
wherein the determining ideal flows further involves a 
difference function associated with flows, home potentials, 
and neighbor potentials. 

28. The network traffic director method of claim 25 
wherein the merit or penalty functions are at least locaUy 
approximated by a quadratic function of flows. 

29. In a data communication system having network 
traffic, a network traffic director system comprising: 

means for storing home potentials associated with a home 
router module; 

means for receiving and storing neighbor potentials asso- 
ciated with neighbor router modules of the home router 
module; 

means for determining ideal flows based on the home and 
neighbor potentials and optimizing at least one of a 
merit function or a penalty function for random chances 
in demands or topology in the data communication 
system; 

means for determining conservation of flows for the home 

router module; 
means for adjusting the home potentials to approach a 

state where flows are conserved if determined not to be 

conserved; 

means for reporting adjusted home potentials to 

neighbor router modules; and 
means for routing network traffic based on home 

neighbor values. 
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